Real-time observation of fluctuations at the driven-dissipative Dicke phase transition 
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We experimentally study the influence of dissipation on the driven Dicke quantum phase transi- 
tion, realized by coupling external degrees of freedom of a Bose-Einstein condensate to the light field 
of a high-finesse optical cavity. The cavity provides a natural dissipation channel, which gives rise 
to vacuum-induced fluctuations and allows us to observe density fluctuations of the gas in real-time. 
We monitor the divergence of these fluctuations over two orders of magnitude while approaching the 
phase transition and observe a behavior which significantly deviates from that expected for a closed 
system. A correlation analysis of the fluctuations reveals the diverging time scale of the atomic 
dynamics and allows us to extract a damping rate for the external degree of freedom of the atoms. 
We find good agreement with our theoretical model including both dissipation via the cavity field 
and via the atomic field. Utilizing a dissipation channel to non-destructively gain information about 
a quantum many-body system provides a unique path to study the physics of driven-dissipative 
systems. 



Experimental progress in the creation, manipulation 
and probing of atomic quantum gases has made it pos- 
sible to study highly controlled many-body systems and 
to access their phase transitions. This new approach to 
quantum many-body physics has substantiated the no- 
tion of quantum simulation for key models of condensed 
matter physics [TJ [2] ■ There has been increasing inter- 
est in generalizing such an approach to zero-temperature 
or quantum phase transitions away from thermal equi- 
librium, as occurring in driven-dissipative systems [5HH]- 
Amongst the most tantalizing questions is how vacuum 
fluctuations from the environment influence the fluctua- 
tion spectrum at a phase transition via quantum back- 
action. Coupling quantum gases to the field of an opti- 
cal cavity is a particularly promising approach to realize 
a driven-dissipative quantum many-body system with a 
well understood and controlled dissipation channel. A 
further advantage of this scheme is that the dissipation 
channel of the cavity mode can be directly used to inves- 
tigate the system in a non-destructive way via the leaking 
cavity field [9]. 

Combining the experimental setting of cavity quantum 
electrodynamics with that of quantum gases [T0l - fT4] led 
to the observation of quantum backaction heating caused 
by cavity dissipation [T5] [TB] , as well as to the realization 
of the non-equilibrium Dicke quantum phase transition 
|17j . Here, we study the influence of cavity dissipation on 
the fluctuation spectrum at the Dicke phase transition by 
connecting these approaches. We non-destructively ob- 
serve diverging fluctuations of the order parameter when 
approaching the critical point, and find a distinct differ- 
ence with respect to predictions for the closed, i.e. non- 
dissipative system. 

In our experimental system, density wave excitations 
in a Bose-Einstein condensate (BEC) are coupled via a 
coherent laser field to the mode of a standing-wave op- 



tical cavity. For strong enough coupling this causes a 
spatial self-organization of the atoms on a wavelength- 
periodic checkerboard pattern which is a realization of 
the driven-dissipative Dicke phase transition jTTJ [TB] . 
The fluctuations triggering the phase transition are 
atomic density fluctuations. They are generated by long- 
range atom-atom interactions which are mediated by ex- 
change of cavity photons [19]. In the presence of cav- 
ity decay, vacuum fluctuations enter the cavity, interfere 
with the coherent pump laser field, and drive the sys- 
tem to a steady state with increased density fluctuations 
[20l 121] . In turn, the cavity decay offers natural access 
to the system properties via the light field leaking out of 
the cavity, which allows us to measure the density fluctu- 
ations in real-time [§]. Except for the natural quantum 
backaction of this continuous measurement process [35] , 
the system remains unperturbed by our observation. 



SYSTEM DESCRIPTION 



Hamiltonian dynamics 



As described in our previous work \17\ I19j , we place a 
BEC of N atoms inside an ultrahigh- finesse optical cavity 
and pump the atoms transversally with a far-detuned 
standing- wave laser field (Fig. [IK)- The closed-system 
dynamics is described by the Dicke model ([TTJ [TH] and 
SI), 
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H = hcoa' a + fuvoJz H — 7=(S + o*)J x , 



N 



(1) 



where u denotes the detuning between pump laser fre- 
quency uj p and dispersively shifted cavity resonance fre- 
quency w c , and H is Planck's constant divided by 27r. The 
annihilation operator of a cavity photon in a frame rotat- 
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FIG. 1: A. Experimental scheme: A transverse pump 
field (red) couples an excited momentum mode of a BEC 
(blue) to a cavity mode via collective light scattering at 
rate A. The cavity provides a loss channel for the system 
through which photons can escape. Density fluctuations 
are inferred from the detected cavity output field. B. 
Level scheme of the system after elimination of the elec- 
tronically excited atomic states. The ground state of the 
closed coupled system is given by a coherent superposi- 
tion of states \n a ,n,b) with even parity (black symbols). 
Here, n a is the number of intracavity photons and n\, is 
the number of momentum excitations. Decay processes 
drive the system to a steady state which also includes 
an incoherent population of states with odd parity (grey 
symbols), caused by either the decay of a cavity photon 
at rate 2k or the decay of a momentum excitation at rate 
27. The depicted level scheme is restricted to n a < 1. 



strength A oc v P between atomic motion and light field 
can be experimentally controlled via the power P of the 
transverse pump field, and represents the collective two- 
photon Rabi frequency of the underlying scattering pro- 
cess between pump and cavity field (Fig. [T]A) . When A 
reaches the critical coupling strength A cr = ^"° , Hamil- 
tonian (nj) gives rise to a second-order quantum phase 
transition [23j towards a phase characterized by a non- 
zero order parameter (J x ) 7^ 0, and a coherent cavity 
field, (a) 7^ 0. Below the critical point, the system is in 
the normal phase, (J x ) = (a) = 0, where only fluctua- 
tions of the order parameter, ( Jj) 7^ 0, give rise to an 
incoherent cavity field with (a^a) 7^ 0. 

In the thermodynamic limit, the fluctuations of the 
order parameter in the normal phase can be described 
with bosonic creation and annihilation operators w and 
b according to J x — y/N(b + o i )/2 (SI). The interaction 
term in Eq. (fTl) then becomes HX(a + a)){b + W), and 
couples the bare states \n a ,rib) under parity conserva- 
tion of the total number of excitations n a + rib. Here, 
n a is the number of photons stored in the cavity and rib 
is the number of excitations in the momentum mode ipi 
(Fig. fl]). The ground state of the closed, coupled system 
is a two-mode squeezed state [Ml US] with admixtures of 
the even parity states only (|0, 0), |1, 1), |0, 2), . . .). For 
u ^$> ujq, the cavity is almost only virtually populated, 
i.e. the admixture of states with n a 7^ is suppressed 
by ujq/ui. The quantum fluctuations of the Hamiltonian 
system then correspond dominantly to pairs of atoms in 
the excited momentum mode ip±. They are created and 
annihilated by quasi-resonant scattering of a pump pho- 
ton into the cavity mode and back into the pump field 
at a rate — . Towards A cr , the variance ((b + b') 2 ) of the 
resulting density fluctuations diverges, while the gas still 
does not show a density modulation ((b + w) = 0). 



Dissipative dynamics 



ing at w p is given by a. The atomic dynamics is captured 
in a two-mode description, consisting of the macroscop- 
ically populated zero-momentum mode ipo of the BEC, 
and an excited momentum mode "01: carrying in a sym- 
metric superposition one photon momentum along the 
±x direction and one along the ±z direction. This de- 
fines an effective two-level system with energy splitting 
hui a = h 2 k 2 /m 7 where k denotes the optical wavevec- 
tor and m the atomic mass. The atomic ensemble of 
N such two-level systems can be described by collective 
spin operators J x , J y and J z . The expectation value ( J x ) 
measures the checkerboard density modulation which re- 
sults from the interference between coherent populations 
of the two matter wave modes and can be identified as 
order parameter of the phase transition. The coupling 



In the case of the open system, the tiny population 
of states with n a 7^ becomes important. As this de- 
cays via cavity dissipation, the ladder of states with odd 
parity (| 1, 0) , |0, 1), |1,2), |0,3), . . .) is incoherently popu- 
lated (Fig. [I]). The microscopic process corresponds to 
the loss of a cavity photon at one of the mirrors before the 
coherent scattering back into the pump beam can be com- 
pleted. The system will thus leave its ground state and 
irreversibly evolve into a non-equilibrium steady state 
with additional density fluctuations and a constant en- 
ergy flow from the pump laser to the cavity output. The 
variance of the resulting incoherent cavity population 
(a) a) has been predicted to diverge at A cr with a critical 
exponent of 1.0 compared to the closed system exponent 
of 0.5 [201 121] • The depletion of the ground state takes 
place at rate k c s = 2 , 1 ■ k, where k is the decay rate 



of the cavity field [IS]. In the experiment, n c g can be 
tuned and we choose u w 8k, such that the rate of de- 
cay processes is almost an order of magnitude below the 
long-range interaction rate \ 2 /ui. 

The observable in our experiment is light leaking out 
of the cavity. Since for k> wo the cavity field adiabat- 
ically follows the atomic motion, the cavity output field 
provides a sensitive tool to monitor the order parameter 
and its fluctuations in real time (SI, [1]). Checkerboard 
density fluctuations with variance ((b + o*) 2 ) induce a fi- 
nite incoherent population of the cavity field according 

to (a* a) = 2^[x^) ({b + tf) 2 ). Cavity decay amounts 
to a continuous measurement of the intracavity light field 
and causes, due to inherent matter-light entanglement, a 
quantum backaction upon the atomic system 22J. The 
role of the photons leaking out of the cavity is thus two- 
fold: they drive the system to a steady state of enhanced 
fluctuations and reveal real-time information about the 
total density fluctuations. 



RESULTS 
Data acquisition 

Using this concept, we experimentally observe density 
fluctuations of the atomic ensemble in the normal phase 
while approaching the phase transition. We prepare the 
system with TV = 1.6(2) • 10 5 87 Rb atoms at an inter- 
mediate coupling of (A/A cr ) 2 « 0.55 and at a detuning 
of bj = 2ir ■ 10.0(5) MHz. Then, the transverse pump- 
laser power is linearly increased within a data acquisi- 
tion time of 0.8 s to a value slightly beyond the criti- 
cal point. For our parameters, loq = 2-7T ■ 8.3(2) kHz and 
k = 27T- 1.25(5) MHz [IS], the rate K e g at which the steady 
state is approached is 2tt ■ 1 kHz for A = A cr [TSJ . We 
can therefore assume the system to be in steady state 
throughout the measurement. The inset in Fig. [2] dis- 
plays the data of a single experimental run, where we 
monitor the stream of photons leaking out of the cavity 
with a single-photon counting module. From the photon 
count rate r we deduce the intracavity photon number 
n = (r — r^)/2KT] 1 taking into account the measured total 
detection efficiency of r\ =5(1) % and the independently 
calibrated background count rate n,. We observe a pro- 
gressively increasing photon count rate with increasing 
transverse pump laser power, until a steep rise marks the 
transition point to the ordered phase. The exact position 
of the transition depends on the total number of atoms 
which fluctuates by 10 % between repeated experimen- 
tal runs. Therefore, we define a threshold for the count 
rate to detect the transition point (SI). This allows us to 
convert the time axis into linearly increasing coupling. 
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FIG. 2: Mean intracavity photon number n (red sym- 
bols) as a function of coupling. Circles (crosses) indicate 
data in the normal (ordered) phase, the errorbars display 
the statistical error. The calculated expectations for the 
closed system are shown as solid black line. Our open- 
system description (solid red line) includes cavity field 
fluctuations due to the decay of photons and momentum 
excitations (grey dashed-dotted line) and due to the fi- 
nite temperature of the BEC (grey dashed line) , as well 
as a symmetry breaking coherent cavity field (grey dot- 
ted line) . We also show the calculated fluctuations if the 
atomic damping rate 7 would vanish (black dashed line) . 
Inset: The raw data of a single run (red line) is displayed 
together with the measured transverse pump power (grey 
dashed line) as a function of time. The sudden increase in 
the photon countrate clearly marks the transition point. 



Mean intracavity photon number 

We average the signal of 372 experimental runs and ob- 
serve the divergence of the intracavity photon number n 
over three orders of magnitude, ending in a steep increase 
after passing the critical point (Fig. [2]) . We compared the 
measured intracavity photon number with the cavity field 
fluctuations expected from the ground state of the closed 
system |24j . Our data clearly shows an enhanced cavity 
field occupation with respect to the Hamiltonian system 
(solid black line in Fig. [2]). This is in accordance with 
the presented picture that cavity decay increases fluctu- 
ations. Yet, the magnitude of the observed fluctuations is 
well below the theoretical expectation [20] [21] for a cavity 
decay at rate n (dashed black line in Fig. [2| . This indi- 
cates the presence of a further dissipation channel which 
damps out atomic momentum excitations. 



Correlation analysis 

Additional insight into the fluctuation dynamics and 
possible dissipative processes can be gained from a cor- 
relation analysis of the cavity output field. We calculate 



the second-order correlation function for all experimen- 
tal data contributing to Fig. [2] Since the cavity field 
adiabatically follows the atomic dynamics, its second- 
order correlation function g^ 2 \r) oc (a^(r)a^(0)a(0)a(r)} 
is linked to the temporal correlation function of the order 
parameter fluctuations (J 2 (t)J 2 (0)}. The evaluated cor- 
relations as a function of time and coupling are shown 
in Fig. [3j together with cuts for specific coupling val- 
ues. In contrast to a purely coherent cavity output field, 
which would yield a flat correlation function, we ob- 
serve enhanced correlations for short times, followed by 
damped oscillations. The frequency of these oscillations 
agrees with the excitation energy of the coupled system, 
hoj a = hojQ^/l — (A/A cr ) 2 , which softens with increasing 
coupling and tends towards zero at the critical point. 
This shows that the cavity output field indeed carries 
information about the incoherent fluctuations of the sys- 
tem, and is consistent with our previous measurement 
of a mode softening and a diverging response [19]. A 
vanishing excitation frequency corresponds to a critical 
slowing down of the dynamics. Within our measurement 
resolution, however, the system adiabatically follows the 
steady state since the rate of change d/dt(X/X cl -) 2 is only 
a few Hz [26] - 

We attribute the damping of the oscillations in g^ (t) 
to the decay of atomic momentum excitations. This con- 
stitutes an additional dissipation channel caused by colli- 
sional and possibly cavity-mediated coupling of momen- 
tum excitations to Bogoliubov modes of the BEC [2"7ll28| . 
The observed decay rate of g^{r) cannot be explained 
by a finite admixture of the cavity field in the steady 
state, since lo exceeds lo by orders of magnitude in our 
system [213] , 

The oscillations in the second-order correlation func- 
tion exhibit an overperiod which becomes more pro- 
nounced towards the critical point. This indicates the 
presence of a finite coherent cavity field amplitude a = 
(a) which we attribute to the finite cloud size of the BEC 
and residual scattering of pump light at the edges of the 
cavity mirrors [17] ■ Interference between the coherent 
and incoherent cavity field components then causes the 
observed overperiod in the correlation function. 



Quantum Langevin description 

To quantitatively describe our observations, we de- 
veloped a theoretical model based on coupled quan- 
tum Langevin equations |29j capturing the dynamics of 
the driven-dissipative system (SI). Our model explicitly 
takes into account the dissipation of the cavity field at 
rate n, and furthermore a dissipation channel for excita- 
tions in the atomic momentum mode ifii. For simplicity, 
this dissipation channel is phenomenologically modeled 
by a thermal Markovian bath at the BEC temperature 
of 100(20) nK into which excitations in the momentum 
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FIG. 3: Temporal correlations of the cavity output. 
Panel A shows a color plot of the measured second-order 
correlation function <? 2 (t) as a function of time r and 
coupling (A/Acr) 2 . The correlation time increases with in- 
creasing coupling in agreement with the timescale related 
to the lowest excitation energy u> s of the coupled system 
(solid black line). Panel B shows the correlations g 2 (r) 
calculated from the full theoretical model with parame- 
ters £ and 7 adjusted to match the data. The horizontal 
dashed lines indicate values of (A/A cr ) 2 along which the 
data is shown (points) on the lower three panels C to- 
gether with the full theory. 



mode tpi decay at a rate 7 (SI). Due to the softening 
excitation frequency u s , the decay rate 7 is taken as a 
function of the coupling rate A. Our model further in- 
cludes a small symmetry breaking field, which results in a 
coherent cavity field amplitude a already below the crit- 
ical point. This is taken into account by renormalizing 
the order parameter with a constant offset £ in Eq. (fTl) 

From the solution of the quantum Langevin equations 
in the thermodynamic limit we obtain the second-order 
correlation function of the intracavity field in the steady 
state (SI). The free parameters of our model description 
(C and 7) are extracted from fits of the model to the 
correlation data (Fig. [3]). We obtain an order parameter 
offset £ — 60(7) at A = 0, which corresponds to 0.8%o of 
the maximal possible order parameter N/2. This agrees 
with our earlier investigation of the symmetry breaking 
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FIG. 4: Damping rate 7 (symbols) as a function of cou- 
pling (A/A cr ) 2 deduced from the cavity output correla- 
tion data. Errorbars indicate the statistical error de- 
rived from the fit. Open circles indicate the region above 
(A/A cr ) 2 = 0.97, where our theoretical model deviates 
significantly from the data and values for 7 might be 
inaccurate. The solid line shows the phenomenological 
function used to model the data in Fig. [2j\. The dashed 
line shows the vanishing excitation frequency lu s /2tt of 
the system. 



field [IS]. 

The extracted damping rate 7 is displayed in Fig. HI 
as a function of coupling. It increases with increasing 
coupling, until it exhibits a cusp around 95% of the crit- 
ical coupling and vanishes towards the critical point. We 
attribute this behavior mainly to the softening of the ex- 
citation frequency uj s (dashed line in Fig. [4l [19]) which 
influences the density of states into which the momentum 
excitations in mode ipi can decay At the critical point, 
this is expected to lead to the absence of damping of the 
excited momentum mode |27l 130] . 

Our model describes our data very well for coupling 
values up to (A/A cr ) 2 ~ 0.97. Above this value, we ob- 
serve enhanced correlations for small r which are not 
captured by the model, as can be seen in the uppermost 
subpanel of Fig. |3p. We believe that in this region tech- 
nical fluctuations, the dynamical change in the dispersive 
cavity shift [31], finite- TV effects [35], and population of 
higher order momentum states start to play a role. 

Using the extracted atomic damping rate and symme- 
try breaking field magnitude, we find very good agree- 
ment between the observed intracavity photon number 
and our model (Fig. [2| . The inclusion of atomic damp- 
ing is crucial for the quantitative description. While 
cavity decay is expected to lead to a strong increase of 
the density fluctuations, atomic dissipation dominantly 
damps out these momentum excitations, such that the 
total fluctuations in the steady state are only moderately 
enhanced with respect to the ground state fluctuations. 
Except for a small region close to the critical point, the 
dominant contribution to the observed fluctuations orig- 
inates from vacuum input noise associated with dissipa- 
tion via the cavity (grey dashed-dotted line in Fig. [2]). 
Only close to the critical point, fluctuations from the 



thermal atomic bath are predicted to contribute because 
the energy of the relevant mechanical excitation vanishes 
towards the phase transition [j"9] . 



Density fluctuations 

We infer the variance of the density fluctuations ((6 + 
w) 2 ) in the normal phase by rescaling the intracavity 
photon number n after subtracting the coherent fraction 
\a\ 2 , which was deduced from the correlation analysis 
(Fig. [5]) . Due to the uncertainties in the symmetry break- 
ing field C, this procedure results in systematical uncer- 
tainties of the deduced density fluctuations which are re- 
flected in the presented error bars. Our data, displayed 
on a log-log scale, deviates clearly in both magnitude and 
scaling from the expectations for the closed system. A 
linear fit (blue line) to the data results in an exponent 
of 0.9(± 0.1). Scaling with exponent 1.0 was predicted 
from open-system calculations in which only cavity dissi- 
pation is taken into account [20"l,l21j. The influence of the 
additional atomic dissipation rate 7 on the scaling of the 
atomic density fluctuations depends on the precise scal- 
ing of this damping rate when approaching the critical 
point, which goes beyond the scope of this publication. 
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FIG. 5: Variance of the checkerboard density fluctuations 
of the BEC, deduced from the intracavity photon number 
after subtracting the coherent contribution. For compar- 
ison, we show the theory of the closed system (black line) 
which diverges with a critical exponent of 0.5, and a lin- 
ear fit (blue line) to the data for (A/A cr ) 2 > 0.9 that 
results in an exponent of 0.9 ± 0.1. We also plot the ex- 
pected fluctuations for a BEC without coupling to the 
cavity field (black dashed line) . The horizontal errorbars 
indicate the statistical error, while the vertical errorbars 
result from the uncertainty in the subtracted coherent 
field component (SI). 



DISCUSSION 



From a more general perspective, driven systems, cou- 
pled via a dissipation channel to a zero-temperature 
Markovian bath, are expected to resemble classical crit- 
ical behavior and can then be characterized in steady 
state by an effective temperature which depends on the 
considered observable (H]-® ESI- In our system, the 
zero-temperature bath is provided by the optical vac- 
uum modes outside the cavity. Verifying the fluctuation- 
dissipation theorem for the order parameter in our sys- 
tem would allow to determine its effective temperature. 
The theoretical expectation of a critical exponent of 1.0 
[20), [21] is a further indication that systems undergoing 
a driven-dissipative phase transition can be described to 
be effectively thermalized. However, answering the ques- 
tion whether cavity dissipation completely destroys the 
quantum character of the system, e.g. the entanglement 
between atomic and light fields, remains a challenge for 
future experiments [20l [29] . 



CONCLUSION AND OUTLOOK 

We have demonstrated the direct observation of di- 
verging density fluctuations in a quantum gas undergo- 
ing the driven-dissipative Dicke phase transition. This 
experiment opens a route to study quantum phase tran- 
sitions in open systems under well controlled conditions. 
Our method directly uses the cavity dissipation chan- 
nel to obtain real-time information on the fluctuations of 
the order parameter. In a similar way, intriguing quan- 
tum many-body states with long-range atom-atom in- 
teractions and the influence of dissipation on them can 
be investigated e.g. by using multi-mode cavities which 
allow to realize glassy and frustrated states of matter 
[551 [31] ■ Adding classical optical lattices to the system 
would let the energy scale of contact interactions enter 
the dynamics and should allow the exploration of rich 
phase diagrams [351 G23 • 
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Supplementary Information 



Experimental details and data analysis 



The experimental sequence for the measurements presented is as follows. After preparing an almost pure Bose- 
Einstein condensate in a crossed-beam dipole trap which is centered with respect to the TEM o cavity mode, the 
transverse pump power P is increased over 50 ms to a relative coupling strength of A 2 /A^ r ~ 0.55. Subsequently, 
the power P is linearly increased over 0.8 s to slightly beyond the critical pump power, while the stream of photons 
leaking out of the cavity is recorded on a single-photon counting module. Intracavity photons are detected with an 
efficiency of 5%, limited mainly by the losses in the cavity mirrors and the detection efficiency of the single-photon 
counter. We carefully calibrated the background count rate of the photon detector in the absence of atoms yielding a 
maximum rate of 341(62) /s. This results from dark counts (at a rate of 100(10)/s) and stray light of the transverse 
pump beam and the dipole trap beams. The background signal is subtracted from the overall count rate recorded in 
the presence of atoms to obtain the mean intracavity photon number, as shown in Figure 2. 

A steep increase of the photon count rate indicates the phase transition point and allows to convert the recording 
time axis into a relative coupling axis A 2 /A^ r . To this end we identify the timebin of 100 /is length in which the photon 
count rate exceeds for the first time the value of 18 /is" 1 corresponding to a mean intracavity photon number of 
approximately 49. According to the typical risetime of the intracavity signal, the critical point is denned as the time 
1 ms prior to this timebin. The relative error of A cr according to this procedure is given by 5 • 10~ 4 . Residual atom loss 
of 10% during probing is taken into account by rescaling the relative coupling axis according to the proportionality 
P C r oc N~ 1 . This is justified by the resulting match between the oscillation frequency of the second-order correlation 
function and the softening excitation frequency, which was measured independently in |Slj . 

For the correlation analysis, the recorded photon data is cut into subtraces whose length decreases to a minimal 
value of 4 ms as the critical point is approached. For each subtrace the photon-photon correlation function is computed 
(using time bins of 2/is). The normalized correlation functions are finally averaged over all 372 experimental runs. 

The temperature of the initially prepared Bose-Einstein condensate was determined from absorption images to 



T = 65(20) nK. During probing, this value is expected to increase by 35 nK due to spontaneous emission and decay 
of atomic momentum excitations into the bath of Bogoliubov modes (see below). Further experimental details can be 
found in EHEEl. 



Theoretical model 
Two-mode description and Dicke model 

We model the dynamics of the transversally driven BEC-cavity system in a two-mode description which formally is 
equivalent to the Dicke Hamiltonian, as was shown previously IS21IS3J . The relevant motional states which are coupled 
via two-photon transitions between pump and cavity fields are given by the macroscopically populated condensate 
mode "00 an d the motionally excited mode ipi which carries in a coherent superposition momenta (p x ,p z ) — (±^fc, ±hk) 
along the cavity (x) and pump (z) directions. To first order, these matter wave modes are separated in energy by 
hujQ = 2hjjjR, with recoil frequency lur — hk 2 /2m, optical wavevector k = 2ir/\ and atomic mass m. Atomic s-wave 
scattering and a weak A/2-periodic lattice potential caused by the transverse pump field induce a small shift of Hujq 

EHU. 

After expanding the atomic field operator in this two- mode basis, "J = coV'o + Ciipi, and introducing corresponding 
collective spin-iV/2 operators (where N denotes the total number of atoms) 

J x \ ( Kqco + cjci) \ 

i} = [ U^-tlh) , (2) 

the many-body Hamiltonian of the BEC-cavity system is given by the Dicke Hamiltonian 

H/h = ua)a + ujQJ z + 2\/\fN{a + a))J x . (3) 

Note that the time-dependence of the driven system was formally eliminated by moving into a reference frame rotating 
at the pump laser frequency ui v . In equation ([3]), a annihilates a photon in the cavity mode with resonance frequency u> c , 
which in the experiment is blue-detuned from the pump laser frequency w p by the amount u) = lu c — lu p = 2tt x 10 MHz. 
Here, the dispersive cavity shift caused by the bare condensate (2tt x 4 MHz for N — 1.6 x 10 5 atoms) is already 
included in u> c . In the non-organized phase (A < A cr ), the change of the dispersive cavity shift due to the small 
population of the motional excited state ipi can be neglected for our experimental parameters (see however |S5US4| ). 
Working at large pump-cavity detuning, we also neglect the substructure of the atoms-shifted cavity resonance caused 
by the polarization dependence of the dipole transition strengths and the cavity birefringence. The coupling strength 
2X/\/N is given by the two-photon (vacuum) Rabi frequency between pump and cavity fields |S2j . 

Residual driving of the cavity field due to finite-size effects of the atomic cloud [S6J and residual scattering of pump 
light off the cavity mirrors results in a finite coherent cavity field amplitude already in the normal phase. The presence 
of this field breaks the Z 2 symmetry of the Dicke Hamiltonian and is modeled by the replacement J x — > J x + C in 
equation (pi, where C (assumed to be real) determines the effective cavity drive amplitude. Effectively, £ can be 
interpreted as an atomic population imbalance between even (cos(fcx) cos(fcz) = 1) and odd (cos(fcx) cos(kz) = —1) 
lattice sites at zero coupling strength |S6) . Since in the experiment C is likely to change in sign and amplitude 
between different experimental runs (e.g. due to drifts of the atomic cloud center with respect to the pump-cavity 
mode structure), we include in the theoretical model an ensemble average according to £ —¥ £cos(0) where <j> is taken 
to change randomly between and 2tt. 



Semi-classical steady-state solution 

The self-consistent mean-field solution for a = (a), (3 = (J_) = (J x — iJy) and w = (J z ) in the presence of 
a symmetry-breaking field (Q ^ 0) is determined from the semi-classical equations of motion of the Hamiltonian 



H + 2h\/\/N(a + a^)C, including cavity decay at rate k: 



Using the conservation of J 2 , i.e. w 2 



-(« + iu)a - i-^tf + P* + 2C) 

v/iV 



8 = —ioJnB + 2i^=(a + a*)w 
y/N 



w = i^=(a + a*)(p-p*). 

■\/N 



N 2 /4, yields the steady-state equations 



A 2 / B 2 

^a^W 1 " 4 ^ 

2A 



(4) 
(5) 
(6) 

(7) 



-(3 + 0/VN 



with critical coupling strength A cr = 



(k 2 +U 2 )uq 



, valid for ( = 0. In Fig. SI, the numerically obtained steady-state 



solution (a, /3, w;) is plotted for typical experimental parameters. 
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FIG. SI. Steady-state solution as a function of relative interaction strength A 2 /A 2 r for our experimental parameters 
and an order parameter offset of £ = 65 (solid curves). The dashed curves display for comparison the case where no 
symmetry-breaking field is present, i.e. £ = (only one branch of the bifurcation is shown). In the experiment, the 
intracavity photon number \a\ 2 rises much faster for A > A cr as compared to the two- mode model since the dispersive 
shift of the cavity frequency dynamically increases with the emerging atomic density modulation|S2j. 



Thermodynamic limit and quantum Langevin equations 

In the thermodynamic limit, N ^> 1, fluctuations of the system around the semiclassical steady-state are small and 
can be treated in a linearized approach |S7j . This is based on the Holstein-Primakoff transformation which maps the 
collective spin degree of freedom J to a bosonic mode with mode operators b and b* , defined as [S8] 



J_ =b\J N -tfb 
J z = Pb - N/2 . 



(8) 
(9) 



Inserting this transformation into the Dicke Hamiltonian, Eq. Efy yields to leading order in N 1 the quadratic 
Hamiltonian 



H'/h = LuStfda + ui dP8b - n(8b + Sb r f + X(Sa + Sa?)(6b + SP) 



(10) 
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with fluctuation operators 5a = a — a and 



b — P/y/N and renormalized parameters 
Re(a)/3 



wo = cjo — 2A 



N 3 / 2 y/l~P 2 /N 2 

2p 2 /N 2 



/i = —A 



V~l - P 2 /N 2 
Re(a)/3 



N 3 / 2 y/l-f3 2 /N 2 



(11) 
(12) 
(13) 



For our experimental parameters and A < A cr , the maximum relative deviation of the renormalized parameters ojq and 
A from their bare values loq and A are 0.3% and 0.7%. For simplicity, we therefore replace in the following analysis 
(A, wo) by the bare values (A, usq). The squeezing term n(5b + 8w) 2 is also neglected in the following as /i/^o < 2 x 10~ 3 
for A < A cr . 

We model the driven-damped system dynamics with coupled quantum Langevin equations of the form [S7. S9lfS10j 



5a = — i[5a, H'\ — nSa + v 2«;ai n (i) 
5b = -i[6b, H'] - j5b + y/Zyb in (t). 



(14) 
(15) 



In equation ( 14 ) , the bosonic operator fii n describes vacuum input fluctuations of the surrounding electromagnetic 



field modes which are characterized by the correlation functions (a,i n (t)al n (t')) — 5(t — t') and (a[ n (t)a- m (t')) = 0. 

In contrast to |S71 IS91 ISlOj . we phenomenologically include an additional dissipation channel for the collective 
atomic motional degree of freedom with effective damping rate 7 accompanied by atomic input fluctuations described 
by 6j n . The physical origin for this dissipation channel is attributed to collisional or cavity- mediated coupling of 
excitations of the momentum mode i\)\ to Bogoliubov modes with wave vectors that are different from that of the 
pump and cavity fields [Sllj . The collection of these modes provides a heat bath at the condensate temperature T. 
For simplicity, we assume this heat bath to behave Markovian. Correspondingly, we require the correlation functions 
for the atomic input noise in Fourier space to obey the relations 



(b in (v)bl(v')) = 27(1 + n th (v))6(v - v 1 ) 
(bl n (p)b- m (v')) = 2 inth {v)5{y - 1/) 



(16) 

(17) 



with thermal mode occupation number nth(^) 



y— P In the data analysis, the damping rate 7 is taken as 



exp(hu/k B T)- 

free parameter which varies as a function of the relative coupling constant A/A cr . This corresponds to the expectation 
that the density of states of the atomic heat bath, evaluated at the softening excitation frequency uj s of the coupled 
system, changes when approaching the phase transition and vanishes at the critical point. 

Due to the coupling between atomic and optical fields, the lowest lying excited states of the coupled BEC-cavity 
system acquire a finite damping rate even for 7 = due to a finite admixture of the cavity degree of freedom. However, 

the corresponding damping rate xz~ K sT+^2 < 27r x 1Hz is negligible for our parameter regime, lu ^> Uo, and does not 
explain the observed damping of the cavity output correlation function. Therefore we expect an additional atomic 
dissipation channel to be present in the system. 



We solve the coupled Langevin equations (14) and (151 in Fourier space according to the conventions 



0{v) = -= / e^O(t)dt 

V Z7T J-00 



<5t(- 



/2vr J- 



e ivt 6\t)dt 



(18) 
(19) 



for any given operator O. We thus obtain the set of coupled equations 



M(i/) 



/ 5a{v) \ 

8tf(-v 

Sb(u) 

\SbH-u)J 



I v2Ka in (^) \ 



2«^(-i/) 

2"fb in (v 



(20) 
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with the 4x4 matrix 



M(v) = 



(—iv + iui + k iX iX 

—iv — ioj + k —iX —iX 

iX iX —iv + iujQ +7 

V — iX —iX —iv — iluq + 7 / 



We note at this point that the cavity input noise operator di n (v) does not vanish for negative frequencies v in 
the frame rotating at the pump laser frequency uj p . This causes the cavity environment to act e.g. on the field 
quadrature a + a 1 ' effectively like a thermal bath |S12j , and results in a distinct change of the fluctuation spectrum of 
the driven-damped Dicke model with respect to the ground state of the Dicke Hamiltonian. 

The Langevin equations in Fourier space are solved by matrix inversion. Denoting the matrix elements of M _1 by 
m,ij(y), we obtain e.g. 



with matrix elements 



and 



6a(v) = V2K(mn(v)ai n (i/) + m 12 (v)al n (-v)) 

+ \/^l{m 13 {v)b- m {v) + mi4(v)bl n (-v)) , 



D x mu(v) = 2iX 2 u>o — i{in + u + ^)((7 — iv) 2 + id 2 ) 

D x mi2(v) = 2iX ujq 

D x m\s{y) = iX(in + 10 + v)(ij + v + u>o) 

D x rdu(i') = iX(in + 10 + v)(ij + v — uq) 



D = ((7 - iv) 2 + Uq) ((k - iv) 2 + u 2 ) - 4X 2 ujuj 



(21) 



(22) 

(23) 
(24) 
(25) 



(26) 



denoting the determinant of M. Analytical expressions for the overlap integrals defined below can be obtained for 
our experimental parameter regime, uj q <Si u, by approximating the determinant D as 



D^{( 1 - t v) 2 +u J 2 )( K 2 +u 2 ) 



(27) 



with the atom- like polariton frequency (soft mode frequency) defined as uj s — loq yl — ^ 2 /-^cr (S lj . This approximation 
eliminates poles of the matrix elements rriij(v) around v = ±lo which do not contribute to the critical behavior of the 
coupled system. 

According to input-output theory, the cavity output field, a ou t{t), is given by 



Oout(*) = v / 2^(a + 5a(t)) - a- m (t). 



(28) 



Correlation functions of the intracavity field 

First-order correlation function 



From (21) we obtain, using the correlation functions of the input operators, 

{h\v)a{v')) = 2n\m 12 (v)\ 2 S(v - v') 

+ 2 1 \m 13 (v)\ 2 n th (v)5(v-v') 

+ 2 1 \m 1A {v)\ 2 (l + n th (~v))S(v - v') 

+ 2rr\a\ 2 6(v)6{v'). 

In the time domain, the first-order correlation function is given by 



GU(T = t-t') = {aHt)a(t')) = ± 

Z7T 



00 /*oo 



/nt 



o t (i/)5(i/))e 



f ~\\ Jivt—iv't' 



AvAv' . 



(29) 



(30) 



OO •/ —OO 
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Inserting Eq. ( 29 1 , this reduces to 

G (D (r ) = 1 A{j ) + l((l + nth(w.))B(r) + n th (io s )B*(r))) + \a\ 2 , (31) 

where we defined overlap integrals 

A(t) = r \m 12 (v)\ 2 e lVT Av (32) 



and made the approximation 



<J — oo 

\m u (v)\ 2 e ivT dv = / \m lz {v)\ 2 e~ lUT Av (33) 

-oo JO 

/-0 

|mij(-^)| 2 "th(^)dz/« nth(w B ) / |mjj(i/)| 2 di/. (34) 

-co >^— oo 

This simplification is justified by the fact that the matrix elements mij(v) peak at ±w«. 

Second-order correlation function 

To calculate the second-order correlation function G^(t — t 1 ) = (a) (t)aJ (t')a(t')a(t)} , we use the relation [S13 : 

G^(t - t') = |G«(0)| 2 + |G«(t - t')\ 2 + \(a(tW))\ 2 - 2|(a(i))| 4 , (35) 

and obtain 

G(2)(r) = |G (1) (0)| 2 + |G (1) (r)| 2 - 2|a| 4 + (36) 

~C(t) + - ((1 + n th (oj s )D( T ) + n th (u> s )D-(r))) + £\a\ 2 * 
with £ = "2~?^2 ) D~(t) = £B*(t) and matrix overlaps 

/oo 
mi 2 (^)mi 1 (-^)e ir/T d^ (37) 

-oo 
/oo 
mi 4 (^)m 13 (-i/)e ll/T dz/. (38) 

-oo 

Uncorrelated background light hitting our detector as well as detector dark counts (corresponding effectively 
to a mean-intracavity photon number of rig) reduce the contrast of the oscillations in the measured second- 
order correlation function. To account for this, we define the normalized second-order correlation function as 

9 < " 2 \ T ) — ig( 1 Uo)+k p " B ' wmcn is in very good agreement with our data upon including the measured back- 
ground count rate. 

Determination of density fluctuations from the cavity output signal 

As the lifetime of cavity photons 1/(2k) is negligible compared to the timescale of atomic motion (determined by 
wo) 7 we c & n directly infer from the detected cavity output field about the magnitude of atomic density fluctuations 



present in the system. The adiabatic solution of equation (14) reads 



Sa(t)= X(6b + 5bt) +V2 K [ t e~^ + ^-<')a in W. (39) 

IK-UJ J_ <x 

Correspondingly, the incoherent intracavity photon number (Sa^Sa) reflects directly the variance of the atomic order 
parameter (i.e. the variance of the checkerboard density modulation): 

(J 2 ) = ^((Sb + Sp) 2 ) = %^(Stf6a) (40) 

as WnW^inW) = 0- After subtracting the coherent fraction \a\ 2 from the total detected intracavity photon number, 
we thus are able to extract the variance of atomic density fluctuations (Figure 5) . 
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Comparison between experimental data and theoretical model prediction 



From least-square fits of our theoretical model to the correlation data (as shown exemplarily in Figure 3C), we 
deduce the decay rate 7 as a function of A 2 /A 2 r (as shown in Figure 4) and a maximum strength of the symmetry 
breaking field given by £ = 60(7). This corresponds (for A = 0) to a finite order parameter of 0.8%o relative to the 
maximum order parameter of N/2 = 8x 10 4 . Since we observe enhanced fluctuations for A 2 > 0.97A 2 r which are not 
explained by our model, we deduce ( from those correlation data where A 2 < 0.97A 2 r . 

We fit to the deduced 7 values the empirical function f(x) = ci(l — x) C2 exp (c^x) + Ci(l — x) C5 exp (cqx) with 
x = A 2 /A 2 r and free parameters Cj. The obtained functional dependence 7(A) as well as the deduced value of £ enter 
the theory curves plotted in Figure 2. 

In this analysis, the temperature of the atomic heat bath was set to the expected temperature T — 100(20) 11K of 
the gas after probing. This is justified by the fact that the thermal population of the momentum mode ip\ becomes 
relevant only close to the transition point due to the softening of the excitation spectrum. 



Error analysis 



The systematic uncertainty of the intracavity photon number (estimated to ±20%) is dominated by the uncertainty 
of the detector quantum efficiency, the cavity mirror transmission and the calibration of the detector background count 
rate. The statistical uncertainty in the determination of A/A cr originates from power fluctuations of the transverse 
pump beam and residual dipole trap oscillations of the atomic cloud, which convert into fluctuations of the critical 
pump power. The uncertainty of £ converts into an uncertainty of the deduced atomic density fluctuations ((b + b') 2 ), 
as indicated by the errorbars in Figure 5. 
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